Foundations of Data Visualization by Unknown

Author:Unknown
Language: eng
Format: epub
ISBN: 9783030344443
Publisher: Springer International Publishing

8.3 Concluding Remarks

The empirical evaluation of the visualization research is far from trivial. As we described above, there are many facets that should be taken care of. We have to select an elevation method, then find appropriate users, correctly execute the evaluation, and present results properly. We agreed that we do need evaluation, and, at the same time, it is clear that it is possible to have a valuable and innovative visualization paper without a user study. Finding the proper evaluation method might be tricky.

As not all members of the visualization community have a proper training in performing studies, many visualization researchers and reviewers often colloquially referred to a controlled laboratory experiment as a user study, while many others consider any empirical study involving the actual users is a user study. These two definitions not only are quite different but also have a limited amount of overlapping. A controlled laboratory experiment is typically conducted in the university environment and its participants commonly include a good number of students and university staff. In many applications, a visualization tool or system is designed for a specific group of users who have better knowledge about the data to be visualized and the tasks to be performed. Although it is possible to design a controlled laboratory experiment with domain experts as participants, this approach is not commonly used because (i) it is difficult to design a set of stimuli for complex scenarios, (ii) the variation of users’ expertise typically becomes a confounding effect, and (iii) the users may find performing tasks in a controlled setting time-consuming or somehow patronizing. Unlike controlled laboratory experiments or semi-controlled crowdsourcing studies, evaluation with domain experts is difficult to reproduce since others cannot easily replicate the same real-world settings, application-specific tasks, and domain experts with similar knowledge. Nevertheless, the lack of reproducibility is a naturally occurring feature rather than a shortcoming.

In our opinion, we should strive to better understanding of the evaluation in the visualization community. There is a vibrant subgroup of the community which organizes BELIV workshop at the IEEE VIS conference. As we do not have a common visualization curriculum across all universities, we recommend to include evaluation topic in teaching of visualization whenever possible. The current state of user studies in visualization often seems like something that has to be done in order for a paper to get accepted instead of contributing to the merit of the paper. This is definitely wrong. We should all learn when to use a user study and when some other means of evaluation is more appropriate. An unsuitable user study creates more harm than good to a paper. A proper evaluation enriches the paper and helps in its acceptance, for sure.

Our take-home message is, unless your work is really “so awesome, cool, impressive, compelling, fascinating, and exciting that reviewers, colleagues, users are totally convinced just by looking at your work and some examples,” consider finding a proper means of evaluation. It will certainly make the paper better. And, even if you write only awesome and fascinating papers, do your homework in study of evaluation methods.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.